MEASURING QoE SATISFACTION IN 5G NETWORKS OR HYBRID 5G NETWORKS

ABSTRACT

A system and method for measuring quality of experience (QoE) satisfaction for an application accessing a CSP network is described. A QoE requirement is associated with an application executed on a mobile device that is communicatively coupled to a CSP network. The QoE requirement for the application includes a QoE latency requirement, a QoE bandwidth requirement, and a QoE packet loss rate requirement. An edge-collection module gathers a radio access network (RAN) data set, and a core network (CN) data set that includes a network data analytics function (NWDAF) data set. The QoE network appliance generates a measured QoE score with the RAN data set, the CN data set, and the NWDAF data set. The measured QoE score is associated with the latency measurement, the bandwidth measurement, and the packet loss rate measurement. A subscriber ID is billed when a charging function determines the measured QoE satisfies the QoE requirement.

FIELD

The description relates to a system and method for satisfying a qualityof experience (QoE) requirement with a measured QoE score in 5G networksand/or hybrid 5G networks. More specifically, the system and methodinclude a QoE network appliance that generates the measured QoE score,which is then compared to a QoE requirement for billing purposes—so thatwhen the measured QoE score meets or exceeds the QoE requirement then asubscriber ID is billed.

BACKGROUND

There have been significant changes to mobile devices and the wirelessnetworks that connect to mobile devices. The launch of 3G mobile devicesand 3G wireless networks (in 2001) made it possible for smartphones tosupport streaming videos, surfing the Web, and downloading music. In2010, 4G network were commercially deployed. The 4G networks were 10times faster than 3G, so downloading games and streaming videos could beperformed with minimum buffering and lags.

In 2019, 5G network were commercially deployed internationally. The 5Gnetworks operate on a much higher frequency range than 4G. The peakspeeds for a 4G network are 100 megabits per second for high mobility,e.g., cars, communications and 1 gigabit per second for low mobility,e.g., stationary, communications. The peak speeds for 5G networks are 10gigabits per second.

The increased speed in a 5G network is achieved by using newhigher-frequency radio waves (25-39 GHz) in addition to the existing lowband (600-850 MHz) and medium band frequencies (2.5-3.7 GHz). However,higher frequency radio waves require smaller geographic cells than thelow and medium band frequencies. The industry consortium settingstandards for 5G is the 3^(rd) Generation Partnership Project (3GPP).

With the migration to 5G, there is a strong need for advanced analyticsto also support closed-loop automation. New 5G services or improvedservices will be needed to justify the cost of upgrading a 4G network.Analytics may be used to support the migration to a 5G network, 5Gservices and improved services. Better 5G analytics can provide betterinsights into consumer activities and can show communication serviceproviders (CSPs) how to deliver improved services and monetize them.

The challenge to communications service providers (CSPs) becomes how tointegrate analytics into the network. Currently, analytics are complexbecause of various non-standardized interfaces, and because ofinconsistent data collection techniques across network vendors. Theseconcerns about analytics having non-standardized interfaces andinconsistent data collection techniques may be addressed by the networkdata analytics function (NWDAF), which is defined as part of the 5G Core(5GC) architecture by 3GPP. 3GPP is the standards development body formobile networks.

NWDAF incorporates standard interfaces from the service-basedarchitecture to collect data by subscription or request from othernetwork functions and similar procedures. These standard interfacesdeliver analytics functions in the network for automation or reporting,which solves the challenges related to non-standardized interfaces andinconsistent data collection.

Therefore, it would be beneficial to provide network analytics thatallow a CSP to gain a view of the customer experience from theperspective of each individual customer and view how the networkconditions are impacting customers from the network perspective.

It would also be beneficial to provide an analytics architecture thatcan be integrated with the services supported by the CSP network and theparticular client devices using the CSP network.

Additionally, it would be beneficial to merge radio access network (RAN)performance data with slice-level telemetry from network data analyticsfunction (NWDAF) and deep-packet inspection (DPI) network data.

Furthermore, it would be beneficial to enable a CSP network tounderstand subscriber expectations based on application usage inreal-time or pseudo real-time.

Further still, it would be beneficial to determine whether the CSPnetwork is satisfying the subscriber expectations.

Further yet, it would be beneficial to charge or bill the subscriberbased on the CSP network satisfying the subscriber's expectations.

Still further, it would be beneficial to determine how to resolve CSPnetwork issues that did not meet the subscriber's expectations.

Also, it would be beneficial to provide new 5G services or improvedservices to justify the cost of upgrading a 4G network.

SUMMARY

A system and method for measuring quality of experience (QoE)satisfaction for an application accessing a CSP network is described.The system includes a mobile device executing the application and havingthe mobile device communicatively coupled to the CSP network. A qualityof experience (QoE) requirement is associated with the application. TheQoE requirement for the application includes a QoE latency requirement,a QoE bandwidth requirement, and a QoE packet loss rate requirement. Anedge-collection module gathers a radio access network (RAN) data set,and a core network (CN) data set that includes a network data analyticsfunction (NWDAF) data set.

A QoE network appliance that includes a core compute and storage networkcomponent receives the RAN data set, the CN data set and the NWDAF dataset from the edge-collection module. The QoE network applianceassociates the RAN data set, the CN data set, and the NWDAF data setwith a QoE latency measurement, a QoE bandwidth measurement, and a QoEpacket loss rate measurement. The QoE network appliance determines theQoE requirement with the RAN data set, the CN data set, and the NWDAFdata set. The QoE network appliance generates a measured QoE score withthe RAN data set, the CN data set, and the NWDAF data set. Additionally,the measured QoE score is associated with the latency measurement, thebandwidth measurement, and the packet loss rate measurement.

The measured QoE score satisfies the QoE requirement by comparing thelatency requirement, the bandwidth requirement, and the packet loss raterequirement associated with the QoE requirement, with the latencymeasurement, the bandwidth measurement, and the packet loss ratemeasurement associated with the measured QoE score. A subscriber ID isbilled when a charging function determines the measured QoE satisfiesthe QoE requirement and the charging function, which is communicativelycouple to a billing system, causes the billing system to bill thesubscriber ID.

In one illustrative embodiment, the system includes a deep packetinspection (DPI) data set. The DPI data set is selected from the opensystem interconnection (OSI) group consisting of a network layer, atransport layer, a session layer, a presentation layer, and anapplication layer. The DPI data set is associated with the latencymeasurement, the bandwidth measurement, and the packet loss ratemeasurement. The DPI data set is also used to determine when themeasured QoE score satisfies the QoE requirement.

In another illustrative embodiment, the edge-based collection modulereduces a volume of data from at least one of the DPI data set, the RANdata set, the NWDAF data and the CN data set before communicating thereduced volume of data to the QoE network appliance. The DPI data setmay also determine an optimal bandwidth measurement for a particularapplication, in which the optimal bandwidth measurement is associatedwith the measured QoE score. Additionally, the DPI data set determinesan optimal latency measurement for the particular application, in whichthe optimal latency measurement is associated with the measured QoEscore.

In yet another embodiment, the QoE network appliance forecasts a perdevice network load for a scaling model with at least one of the RANdata set, the NWDAF data set and the CN data set.

In a further embodiment, the QoE network appliance forecasts a perapplication network load for a scaling model with the DPI data set, theRAN data set, the NWDAF data set and the CN data set.

In a still further embodiment, the QoE network appliance forecasts a perlocation network load for a scaling model with the DPI data set, the RANdata set, the NWDAF data set and the CN data set.

In an even further embodiment, the system includes a CSP network policy,in which a reduced network performance is determined with the measuredQoE score at an impacted area, and the CSP network alerts at least onemobile device of the reduced network performance.

In another embodiment, the system includes a CSP network policy thatchanges based on the measured QoE for the subscriber ID at a particularcell.

DRAWINGS

The present subject matter will be more fully understood by reference tothe following drawings which are presented for illustrative, notlimiting, purposes.

FIG. 1A shows an illustrative radio access network (RAN) system thatprovides a smartphone with Internet connectivity.

FIG. 1B shows data inputs and outputs to an OPS-IQ software module.

FIGS. 2A, 2B and 2C show a network analytics architecture that includesNWDAF operating as a network element and a data collection source.

FIGS. 3A and 3B show an illustrative high-level network analytic billingsystem that charges subscribers based on the QoE.

FIG. 4 shows a block diagram having various components for an eventbased billing mediation system 400, which operates with the systemsdescribed in FIG. 1 through FIGS. 3A and 3B.

FIGS. 5A, 5B and 5C show various methods for communicating applicationQoE requirements and the integration of the QoE requirements withapplications.

FIGS. 6A, 6B, and 6C show an illustrative flowchart for determining aQoE requirement, measuring satisfaction of QoE, billing for QoE,resolving unsatisfactory QoE.

FIG. 7 shows a method for generating an integrated event stream that iscommunicated to a robotic process automation (RPA) module.

DESCRIPTION

Persons of ordinary skill in the art will realize that the followingdescription is illustrative and not in any way limiting. Otherembodiments of the claimed subject matter will readily suggestthemselves to such skilled persons having the benefit of thisdisclosure. It shall be appreciated by those of ordinary skill in theart that the systems and methods described herein may vary as toconfiguration and as to details. Additionally, the methods may vary asto details, order of the actions, or other variations without departingfrom the illustrative methods disclosed herein.

The systems and methods described herein can be used to support amigration to 5G networks, 5G services and improved hybrid services.Additionally, the systems and methods described herein include networkanalytics that provide better insights about subscriber usage and show acommunications service provider (CSP) how to deliver improved servicesand monetize them. Furthermore, the systems and methods described hereinovercome the CSP challenges of integrating network analytics into theCSP network. Further still, the systems and methods described hereinovercome the complexity associated with various non-standardizedinterfaces and inconsistent data collection techniques across networkvendors.

The systems and methods described herein allow a CSP to understand thecustomer experience in real-time and view how the network conditions areaffecting the customer. Additionally, the systems and methods provide ananalytics architecture that can be integrated with the servicessupported by the CSP network and the particular client devices using theCSP network. Furthermore, the systems and methods described herein mergeradio access network (RAN) performance data with slice-level telemetryfrom network data analytics function (NWDAF) and deep-packet inspection(DPI) network data. Further still, the systems and methods enable a CSPto view the network resources utilized by subscribers based on thenetwork slice architecture, a cell site, a device type, or a customersegment.

A Radio Access Network (RAN) is part of a telecommunication system thatutilizes a Radio Access Technology (RAT). The RAN resides between UserEquipment (UE) and provides a connection to a Core Network (CN). A basestation is related to a site coverage area and may include a cell site,a sector, a frequency or any other parameter associated with the RANsite that may be monitored, controlled or the combination thereof. UserEquipment (UE) includes devices such as a smartphone, mobile phones, acomputer, an IoT device, and other such devices. Radio AccessTechnologies (RATs) refers to the underlying physical connection methodfor a radio-based communication network. For example, a smartphone maycontain several RATs such as Bluetooth, Wi-Fi, 3G, 4G, LTE and 5G. Dataflow is measured in bytes per second.

The network analytics described herein are used to deliver new servicesthat leverage 3GPP standards based architecture for 5G networks andhybrid 4G/5G networks. The term “hybrid 4G/5G networks” is also usedinterchangeably to refer to a “hybrid 5G network.” The network analyticsdescribed herein also provide an ability to determine a Quality ofExperience (QoE) requirement and make it a billable asset for the CSP.For example, the QoE requirements may be determined with Radio AccessNetwork (RAN) data, Deep Packet Inspection (DPI) data, and Network DataAnalytics Function (NWDAF) data.

The systems and methods presented herein enable the CSP to offerdifferentiated services based on user experience and bill based on theCSP delivery of a Quality of Experience (QoE). QoE refers to a measureof the customer experience with a service, e.g., streaming video,gaming, phone call, TV broadcast. QoE focuses on the entire serviceexperience and is a holistic concept. More specifically, QoE is definedas the degree of delight or annoyance of the user with an application orservice—and QoE results from the fulfilment of his or her expectationwith respect to the utility and/or enjoyment of the application orservice in light of the user's personality and current state.

QoE is distinguishable for Quality of Service (QoS). QoS is thedescription or measurement of the overall performance of a service, suchas a telephony or computer network or a cloud computing service,particularly the performance seen by the users of the network. Toquantitatively measure quality of service, several related aspects ofthe network service are often considered, such as packet loss, bit rate,throughput, transmission delay, availability, jitter, and other suchnetwork analytics.

To effectively determine the QoS in a 5G network, the CSP must haveaccurate network analytics that constantly determine how a subscriberuses the network, the subscribers experience, how the network deliverspackets to the subscriber, and the cost of continuously deliveringpackets.

Network analytics are commonly used for QoS purposes. Network analyticscan also be used for QoE purposes to provide a deep understanding of howthe CSP network is delivering packets to the customer. The systems andmethods described herein monitor each session, from the applicationtype, network slice loading, and RAN resource utilization andperformance to provide true visibility into the QoE. The systems andmethods described herein utilize network analytics to determine QoE inhybrid 5G and 5G networks. For example, the QoE network analytics mayinclude data sets for Radio Access Network (RAN) performance, RANquality and slice level telemetry from Network Data Analytics Function(NWDAF). These network analytics may be integrated with deep packetinspection (DPI) data sets. These network analytics allow the CSPnetwork to monitor real-time network performance, real-time networkusage, and the QoE for each subscriber.

As described herein, NWDAF is used to perform at least two differenttypes of processes in the systems described herein. The first processperformed by NWDAF is edge collection of data with a NWDAF edgecollector, which is referred to as NWDAF edge collection, which isdescribed in FIG. 2B. The second process performed by an NWDAFapplication is gathering data sets from various elements of the 5G coreas presented, which generates an NWDAF integrated event stream asdescribed with the system elements presented in FIG. 1B and the processsteps presented in FIG. 7 .

Note, the NWDAF integrated event stream only includes NWDAF. A moregeneral reference is also made to an “integrated event stream” thatincludes RAN information, and core network (CN) information, and DPIinformation. In general, the integrated event stream refers to datacaptured by the WAN/data bus associated with a QoE network appliancethat is communicatively coupled to a robotic process automation (RPA)module—as described in further detail below. The integrated event streammay include data sets collected from other edge base collectors. Theintegrated event stream may include network events. Network events mayalso include all requests caused by a user interaction, a user action, anetwork interaction, a subrequest.

NWDAF may also operate as a network element that interacts with avariety of different network elements. In the illustrative embodiment,the NWDAF integrated event stream includes RAN data, CN data, andpossibly DPI, which is used to determine a measured QoE score thatdetermines when the CSP is meeting the QoE requirement.

A low measured QoE score may result in a robotic process automation(RPA) module determining an action that must be taken to improve themeasured QoE score. In this illustrative embodiment, the NWDAF mayoperate as a network element that notifies the subscriber about the QoE—this is referred to as a “subscriber” notification and is described inFIG. 6 . The NWDAF may also operate as a network element with respect toa network operation notification that is also described further in FIG.6 .

The systems and methods presented herein rely on NWDAF for networkanalytics to overcome non-standardized interfaces and inconsistent datacollection techniques. NWDAF incorporates standard interfaces from theservice-based architecture to collect data by subscription or requestfrom other network functions and similar procedures. These standardinterfaces deliver analytics functions in the network for automation orreporting, which overcomes the challenges related to non-standardizedinterfaces and inconsistent data collection.

Analytics are not limited to NWDAF. CSPs monitor and analyze othernetwork resources. For example, the CSP network analytics may analyzethe network from various perspectives such as per slice, per cell site,per device type, or other such network variable. The systems and methodsdescribed herein provide network analytics that allow a CSP to gain aperspective of the customer experience from each individual subscriberand view how the network conditions are impacting subscriber QoE. Thenetwork analytics architecture described herein can be integrated withthe services supported by the CSP network and the particular clientdevices using the CSP network. The systems and methods presented hereinmerge radio access network (RAN) performance data with slice-leveltelemetry from network data analytics function (NWDAF) and deep-packetinspection (DPI) network data.

More specifically, the network data analytics function (NWDAF) allowsnetwork function (NF) consumers to subscribe to and unsubscribe fromdifferent analytic events. Also, this service notifies NF consumers witha corresponding subscription about observed events. The types ofobserved events include load level of network slice instance, serviceexperience for an application or for a network slice, NF load analyticsinformation for a specific NF or a list of NFs, network performance inan area of interest, expected behaviour information for a group of userequipment (UE) or a specific UE, abnormal behaviour information for agroup of UEs or a specific UE, mobility related information for a groupof UEs or a specific UE, communication pattern for a group of UEs or aspecific UE, congestion information of user data in a specific location,and QoS sustainability for a certain area and time period, reports QoSchange statistics or predicts the likelihood of a QoS change. NWDAFprovides analytics information for different analytic events to NFconsumers. NWDAF allows NF consumers to subscribe to and unsubscribefrom periodic notifications and/or notification when an event isdetected.

5G systems are designed based on new network technologies that includenetwork function virtualization (NFV), software-defined networking (SDN)and network slicing. 5G systems support service based architecture (SBA)which allows the network functions (NF) to discover other networkservices and communicate, unlike the older technologies where they hadpredefined interfaces between entities. The service-orientedarchitecture in the 5G system is more flexible, customizable, andscalable. The 5G systems support the stateless network functions wherethe compute resource elements are decoupled from the storage resourceelements.

The Network Slice Selection Function (NSSF) supports taking informationfrom the NWDAF into consideration for slice selection. The NSSF accessesthe NWDAF events subscription service. Network slicing is a specificform of virtualization that allows multiple logical networks to run ontop of a shared physical network infrastructure. The key benefit of thenetwork slicing concept is that it provides an end-to-end virtualnetwork encompassing not just networking but compute and storagefunctions too. The objective is to allow a physical mobile networkoperator to partition its network resources to allow for differentusers, so-called tenants, to multiplex over a single physicalinfrastructure. The most commonly cited example in 5G discussions issharing of a given physical network to simultaneously run Internet ofThings (IoT), Mobile Broadband (MBB), and very low-latency (e.g.,vehicular communications) applications. These applications havedifferent transmission characteristics. For example, IoT has a largenumber of devices, but each device may have low throughput. MBB hasnearly the opposite properties since it will have a much smaller numberof devices, but each one will be transmitting or receiving highbandwidth content. The intent of network slicing is to be able topartition the physical network at an end-to-end level to allow optimumgrouping of traffic, isolation from other tenants, and configuring ofresources at a macro level.

Network slicing in 5G is expected to open new business opportunities formobile operators and other newer entrants. For example, a CSP networkcan split its physical network resources into multiple logical slicesand lease these slices out to interested parties. For example, anelectrical utility may want to have a long-term lease of a network slicefor connectivity to its smart grid composed of sensors, meters, andcontrollers and optimize that slice for IoT devices. Alternatively, aconcert promoter may want to take a short-term lease of a network slicefor a week-long musical festival and optimize that slice for streamingHD music and Voice over Internet Protocol (VoIP) connectivity.

A typical 5G network includes an access network and a core network. The5G access network includes user equipment (UE) supporting a 5G newradio, a base station that supports 5G new radio (gNB) and existing LTEeNodeB upgraded to support 5G new radio. The 5G core network introduceda service based architecture (SBA) that replaced traditional nodes withindividual network functions, which run in a completely virtualizedenvironment. There are various network functions in the 5G core thatinclude an access and mobility management function (AMF), a sessionmanagement function (SMF), a user plane function (UPF), a networkexposure function (NEF), an application function (AF), a policy controlfunction (PCF), a network repository function (NRF), a unified datamanagement (UDM), an authentication server function (ASF), unified datarepository (UDR) and network slice selection function (NSSF).

The 5G system provides a wide range of services and applications thathave different characteristics and performance requirements such asenhanced mobile broadband (eMBB), ultra-reliable and low latencycommunications (URLLC) and massive machine type communication (mMTC).Enhanced mobile broadband (eMBB) are data driven services that providegreater bandwidth with moderate latency such as office productivityscenarios, user upload and sharing multimedia files, remote educationscenarios, and enhanced broadband in fast moving trains and airplanes.Ultra-reliable and low latency communications (URLLC) are provided tomission critical services; and the services are extremely low latencyand high reliability. Power plants, military applications, remotesurgeries, industrial automation, tactile Internet, disaster, andemergency service require very low latency and high reliability. Massivemachine type communications (mMTC) support a large number of deviceswithin a small area, which communicate data sporadically. Illustrativeuse cases include Internet-of-Things (IoT), smart metering and streetvideo recording.

Referring to FIG. 1A, there is shown an illustrative radio accessnetwork (RAN) system 100, e.g. an LTE network, which provides mobiledevices 104, i.e., User Equipment (UE), such as a smartphone withInternet connectivity. Note, reference to UE 104 is also made byreferring interchangeably to a mobile device, a wireless device, aclient, a wireless client and other such more common references to UE.The illustrative mobile device 104 communicates with at least one eNodeB106. The illustrative mobile device 104 may include an InternationalMobile Subscriber Identity (IMSI).

More generally, the illustrative mobile device 104 may include apersonal computer, a laptop, a tablet computer, or a smartphone. Themobile device 104 may be operationally coupled to a wide area network(WAN) such as the Internet by being communicatively coupled to a RadioAccess Network (RAN) associated with a service provider network. Themobile device 104 may also be communicatively coupled to the WAN via aWi-Fi (or Bluetooth) access point (not shown) that is communicativelycoupled to an illustrative modem (not shown), which is communicativelycoupled to the WAN.

In operation, the mobile device 104 downloads an application from an appstore and executes the application on the mobile device 104. The mobiledevice 104 communicates a first message communicated to a QoE networkappliance 121. The first message indicates that the application has beenlaunched on the mobile device. A second message is communicated to theQoE network appliance. The second message indicates that the applicationis closed. In one embodiment, the first message received at the QoEnetwork appliance 121 is associated with a QoE requirement. The QoErequirement that includes a QoE latency requirement, a QoE bandwidthrequirement, or a QoE packet loss rate requirement;

The QoE network appliance 121 may be embodied as a dedicated deviceand/or as a network based service. In one embodiment, the QoE networkappliance 121 includes a stand-alone network device, e.g., a dedicatedserver 120 having a processor 122 and memory 124.

The QoE network appliance 121 may also be embodied as a network basedservice 121, which is also referred to interchangeably as a cloud-basedservice. The cloud-based service may operate as one of four fundamentalcloud service models, namely, infrastructure as a service (IaaS),platform as a service (PaaS), software as a service (SaaS), and networkas a service (NaaS). The cloud service models are deployed usingdifferent types of cloud deployments that include a public cloud, acommunity cloud, a hybrid cloud, and a private cloud.

Infrastructure as a service (IaaS) is the most basic cloud servicemodel. IaaS providers offer virtual machines and other resources. Thevirtual machines, also referred to as “instances,” are run as guests bya hypervisor. Groups of hypervisors within the cloud operational supportsystem support large numbers of virtual machines and the ability toscale services up and down according to customers' varying requirements.IaaS clouds often offer additional resources such as images in a virtualmachine image library, raw (block) and file-based storage, firewalls,load balancers, IP addresses, virtual local area networks (VLANs), andsoftware bundles. IaaS cloud providers supply these resources on demandfrom their large pools installed in data centers. For wide areaconnectivity, the Internet or virtual private networks (VPNs) can beused.

Platform as a service (PaaS) enables cloud providers to deliver acomputing platform that may include an operating system, a programminglanguage execution environment, a database, and a web server.Application developers can develop and run their software solutions onthe PaaS without the cost and complexity of buying and managing theunderlying hardware and software layers. With some PaaS solutions, thesystem resources scale automatically to match application demand, so thecloud end user does not have to allocate resources manually.

Software as a service (SaaS) enables cloud providers to install andoperate application software in the cloud. Cloud end users access thesoftware from cloud clients. The cloud end users do not manage the cloudinfrastructure and platform that runs the application. The SaaSapplication is different from other applications because of scalability.Higher throughput can be achieved by cloning tasks onto multiple virtualmachines at run-time to meet the changing work demand. Load balancers inthe SaaS application distribute work over a set of virtual machines. Toaccommodate a large number of cloud end users, cloud applications may bemultitenant and serve more than one cloud end user organization. SomeSaaS solutions may be referred to as desktop as a service, businessprocess as a service, test environment as a service, communication as aservice, etc.

The fourth category of cloud services is Network as a service (NaaS), inwhich the capability provided to the cloud service end user is to use anetwork/transport connectivity service, an inter-cloud networkconnectivity service, or the combination of both. NaaS involves theoptimization of resource allocations by considering network andcomputing resources as a unified whole. Traditional NaaS servicesinclude flexible and extended VPNs, and bandwidth on demand.

There are different types of cloud deployment models for the cloud basedservice, which include a public cloud, a community cloud, a hybridcloud, and a private cloud. In a public cloud, applications, storage,and other resources are made available to the general public by aservice provider. These services are free or offer a pay-per-use model.

The community cloud infrastructure is between several organizations froma community with common concerns, and can be managed internally or by athird-party, and hosted internally or externally; so the costs arespread over fewer users than a public cloud (but more than a privatecloud).

The private cloud infrastructure is dedicated for a single organization,whether managed internally or by a third-party, and hosted internally orexternally. A private cloud project requires virtualizing the businessenvironment, and it requires that the organization reevaluate decisionsabout existing resources.

The hybrid cloud is a composition of two or more clouds (private,community or public) that remain unique entities but are bound together,offering the benefits of multiple deployment models. Hybrid cloudarchitecture requires both on-premises resources and off-site (remote)server-based cloud infrastructure. Although hybrid clouds lack theflexibility, security, and certainty of in-house applications, thehybrid cloud provides the flexibility of in-house applications with thefault tolerance and scalability of cloud-based services.

Referring back to FIG. 1A, the illustrative radio network system 100provides User Equipment 104 (UE) such as a smartphone with Internetconnectivity. When the mobile device 104 has data to send to or receivefrom the Internet, it sets up a communication channel between itself andthe Packet Data Network Gateway 114. This involves message exchangesbetween the UE 104 and the Mobility Management Entity (MME) 108. ThePacket Data Network (PDN) gateway 114 performs policy enforcement,packet filtering for each user, charging support, lawful inspection, andpacket screening. Additionally, the PDN gateway 114 includes a Policyand Charging Enforcement Function (PCEF) 136, which communicates with aPolicy Charging Rules Function (PCRF) 138. The PCRF is a software nodethat determines policy rules in a multimedia network in real time.Additionally, the PCRF is the part of the network architecture thataggregates information to and from the network, operational supportsystems, and other sources (such as portals) in real time, supportingthe creation of rules and then automatically making policy decisions foreach subscriber active on the network. The PCEF 136 and PCRF 138 areboth communicatively coupled to the QoE network appliance 121.

In coordination with the eNodeB base station 106, the Serving Gateway112, and the Packet Data Network Gateway 114, data plane tunnels areestablished between the base station 106 and the Serving Gateway 112,and between the Serving Gateway 112 and the Packet Data Network Gateway114. The network establishes a virtual communication channel, called anEvolved Packet switched System (EPS) bearer, to connect the UE 104 andthe base station 106.

For network access and service, entities in the illustrative network 100exchange control plane messages. A specific sequence of such controlplane message exchange is called a network procedure. For example, whena mobile device 104 powers up, it initiates an attach procedure with theMME 108, which includes establishing a radio connection to the basestation 106. Thus, each network procedure involves the exchange ofseveral control plane messages between two or more entities. Thespecifications for these are defined by the various 3GPP TechnicalSpecification Groups.

A network interaction includes detecting and monitoring network events.Network events may include all requests caused by a user interaction, auser action, a network interaction and a subrequest. By way of exampleand not of limitation, a network interaction is the set of HTTP requestsand responses, or other sequence of logically related network traffic,caused by a user visit to a single web page or interaction with pageelements. Also, a network interaction may be a single HTTP request andits corresponding responses such as zero or interim responses and singlefinal response. A network interaction may also include a user action,which is a deliberate action by the user, via configuration, invocation,or selection, to initiate a network interaction, selection of a link,submission of a form, and reloading a page are examples of user actions.Furthermore, a network interaction may include a subrequest that is notdirectly initiated by user action. An example of a network interactionthat includes a subrequest that is not initiated by a user action is aninitial response in a hypermedia format that contains embeddedreferences to stylesheets, images, frame sources, and onload actionswill cause a browser, depending on its capabilities and configuration,to perform a corresponding set of automated subrequests to fetch thosereferences using additional network interactions.

The user plane carries the network user traffic. The user plane is alsoreferred to as data plane, carrier plane or bearer plane; the user planecontains data regarding the content that is sent or received by theuser, e.g., text messages, voice, photos, videos and websites.

The control plane carries signaling traffic that is originated from ordestined for a router. Control plane contains the data regardingmanagement of the network. Control plane makes decisions about wheretraffic is sent. The control plane functions include the systemconfiguration, management, and exchange of routing table information.

An illustrative session time having a variable time period may bedefined as “variable” based on the variable duration of voice call, thevariable duration of a gaming session, the variable duration of astreaming video session, or any other such variable time period for userselected data traffic. A fixed time interval period may be a fixed timeinterval during which the network events are monitored in a regularmanner that is determined by a fixed time interval. Also, the sessionmay be a combination of a variable time period and fixed time interval.For example, a video call may include detecting and monitoring the videoquality at fixed intervals and detecting and monitoring the voice callquality at fixed intervals—additionally, voice and video quality may bedetected and monitored for the duration of the video call.

Referring to FIG. 1B, there is shown data inputs and outputs to anOPS-IQ software module. The OPS-IQ software module 152 is a real-time,context-aware, operational analytics powered by machine learning (ML)that ingests and processes alerts, alarms, and other telemetry data fromdisparate sources associated with the communications service provider(CSP). In operation, the OPS-IQ software module operates across multipleoperational domains corresponding to dynamic physical and virtual CSPnetwork infrastructure.

By way of example and not of limitation, the OPS-IQ software module 152may be implemented either as a network element on a dedicated hardwaredevice, as a software instance running on a dedicated hardware device,or as a virtualized function instantiated on an appropriate platform,e.g., on a cloud infrastructure.

The term “network element” as used herein refers to a facility orequipment used in the provisioning of a telecommunication service, andalso includes features, functions, and capabilities that are provided bythe facility or equipment including, but not limited to, subscribernumbers, databases, signaling systems, and information sufficient forbilling and collection or used in the transmission, routing, or otherprovisioning of a telecommunication service.

The OPS-IQ software module 152 addresses CPS network and service issuesthrough real-time ML based operational analytics, topology independentroot issue analysis, fault correlation, fix recommendation, incidentprediction and automatic trouble ticket generation. The OPS-IQ softwaremodule 152 is designed for network teams, service teams, field teams andcustomer care teams. The OPS-IQ software module 152 provides aself-healing network paradigm to prevent customer-impacting servicedegradations and outages. Additionally, the OPS-IQ software module 152lowers operational expenditures and optimizes the overall customerexperience (CX).

In general, the OPS-IQ software module 152 collects and correlates datafrom other network elements and OSS/BSS systems. OSS/BSS refers tooperations support system (OSS) and business support system (BSS),respectively. CSPs support a broad range of services and functions, inwhich OSS includes order management, network inventory management andnetwork operations and BSS includes order capture, customer relationsmanagement, and telecommunications billing.

More specifically, the OPS-IQ software module 152 includes threemodules, namely, a network fault analytics module (not shown), a serviceexperience analytics module (not shown), and a live ops analytics module(not shown). The network fault analytics module addresses alarmmanagement challenges through advanced machine learning (ML) techniquesto understand which alarms are important and most relevant and impactCX. The service experience analytics (SEA) module correlates network andservice degradation issues with CX impact for a CX-first problem-solvingapproach using real-time detection and prediction, as well asprescriptive actions. Additionally, the SEA module provides aself-healing network in a highly dynamic virtualized network environmentdelivering optimized service delivery and subscriber QoE. The LiveOpsanalytics module manages network and customer-premises equipment (CPE)performance issues through time series analysis of operationalnetwork-based events using event data scoring and anomaly detection. TheOPS-IQ software module 152 is a versatile data integration tool thatreceives and processes data from the SEA module and the LiveOps module.Additionally, the OPS-IQ software module 152 may be rules and topologyindependent.

In operation, the illustrative OPS-IQ software module 152 receives 3Gand 4G datasets associated with a legacy system operation,administration, and management (OAM) 154. In the illustrativeembodiment, OAM is a data collection source. OAM is not specified by3GPP but stands for Operations, Administration and Maintenance (OAM),which refers to the processes and functions used in provisioning andmanaging a network or element within a network. The NWDAF/OAM interface156 runs specific scripts to make other applications work better. Thereare two types of OAM. The first type of OAM relates to legacy 4Gnetworks or “transport OAM.” The second type of OAM is related to 5G andcan combine OAM-IP and OAM-5G. Note, with respect to the 5G systems andNWDAF data sets, there is no need for a data collection source, e.g.,OAM, because the data sets are delivered directly. Also, the NWDAF datasets are shared with the OAM data collection, which supports receivingobserved service experience from the NWDAF.

The OPS-IQ software module 152 also receives RAN data 158 from 4Gnetworks. RAN data 158 can be collected from access mobility managementfunction (AMF) 164, open radio access network (O-RAN) and traditionalnetwork equipment providers (NEPs). By way of example, the RAN data 158includes per call trace data having approximately 60 data sources,power, bears and other such trace data. The RAN data 158 can support loganalysis, which further supports maintenance use cases. For example, theRAN data collects information directly from data sources every twominutes, but not in real-time.

Additionally, the OPS-IQ software module 152 receives virtualizednetwork function (VNF) data 160. VNFs are virtualized network servicesrunning on open computing platforms formerly carried out by proprietary,dedicated hardware technology. Common VNFs include virtualized routers,firewalls, WAN optimization and network address translation (NAT)services. In a 5G implementation, VNF is a software implementation of anetwork equipment, such as a router, a firewall, a load balancer, oreven the components that conform to the mobile core network. Forexample, syslog data is received, in which syslog data refers to the useof standard message format to communicate with a logging server. Also,the simple network management protocol (SNMP) may be used to monitornetwork performance.

In the illustrative embodiment, OAM data 154, RAN data 158 and VNF data160 is received and integrated by the OPS-IQ software module 152, whichis shared with an NWDAF application 162. The NWDAF application 162allows data to be collected once and shared among many otherapplications and/or functions. In other words, the data sets that areprocessed by the OPS-IQ software module 152 are then shared with theNWDAF application 162 for enrichment and for NWDAF exfiltration to thenetwork functions described in further detail below.

The NWDAF application 162 combines the 3G and 4G datasets with 5G datasets and the NWDAF data is shared with one or more network functions bythe NWDAF application. By way of example and not of limitation, theNWDAF functions are shared by an illustrative subscription service withthe access and mobility management function (AMF) 164, the sessionmanagement function (SMF) 166, the network slice selection function(NSSF) 168, the policy control function (PCF) 170, the network exposurefunction (NEF) 172, the application function (AF) 174 an OAM eventfunction 176 and other such network functions.

The data set from NWDAF is shared with the access and mobilitymanagement (AMF) function 164. The core AMF functions includeregistration management, connection management, reachability management,mobility management and various functions relate to security and accessmanagement and authorization. With respect to the NWDAF interaction andthe AMF function, AMF subscribes to NWDAF to take session managementfunction (SMF) load information from NWDAF into consideration for SMFselection.

Additionally, the data set from NWDAF is shared with the sessionmanagement function (SMF) 166, which is one of the main functions in thenext generation core. More specifically, SMF includes variousfunctionality relating to subscriber sessions, i.e., sessionestablishment, modification, and release. The NWDAF interaction supportstaking user plane function (UPF) load information from NWDAF intoconsideration for UPF selection. UPF performs user plane operations likemaintaining a session, packet routing and forwarding, packet inspection,policy enforcement for the user plane and other such user planeoperations.

Also, the data set from NWDAF is shared with the network slice selectionfunction (NSSF) 168, which supports local level information from NWDAF,which is taken into consideration for slice selection. The NSSF accessesthe NWDAF subscription service via an illustrative N34 reference point.Note, the N34 reference point is a reference point between the NSSF andthe NWDAF.

Furthermore, the data set from NWDAF is shared with the policy controlfunction (PCF) 170, which supports the unified policy framework thatgoverns network behavior. The PCF provides policy rules to the controlplane functions to enforce them. The subscription information isgathered from the unified data management function. The NWDAFinteraction takes input from NWDAF into consideration for policies onassignment of network resources and for traffic steering policies. Thereis a subscription for analytics information during background datatransfer from the NWDAF to determine whether the negotiated transferpolicy is impacted. The NWDAF also takes analytics informationassociated with Quality of Service (QoS) sustainability from the NWDAFinto consideration for QoS policies.

Further still, the NWDAF data set is shared with the network exposurefunction (NEF) 172, which provides a way to securely expose the servicesand capabilities provided by the 3GPP network functions. The NWDAFinteraction supports forwarding UE mobility information from NWDAF tothe application function (AF) when it is untrusted, forwarding UEcommunication information from the NWDAF to the AF when it is untrusted,and forwarding user data congestion information from NWDAF to the AFwhen it is untrusted.

Further yet, the NWDAF is shared with the application function (AF) 174,in which the application front end (AFE) serves as the lightweightfront-end access to the unified data repository (UDR). For example, fora mobile switching center server (MSC-S) to access subscriber datastored in the UDR, it must communicate with home location registry (HLR)application front end (AFE). Similarly, for the mobile management entity(MME) to access subscriber data, it must communicate with the homesubscriber server (HSS) application front end (AFE). With respect to theNWDAF interaction, the NWDAF interaction supports receiving UE mobilityinformation from NWDAF or via the NEF, supported UE communicationinformation from NWDAF or via the NEF and supports receiving user datacongestion information from NWDAF or via the NEF.

Referring to FIGS. 2A, 2B and 2C there is shown a network analyticsarchitecture that includes NWDAF operating as a network element and adata collection source. In FIG. 1B presented above, the NWDAFapplication was presented. NWDAF may operates as an application, anetwork element, and as a data collection source in a distributedarchitecture.

In FIGS. 2A, 2B, and 2C, the network analytics architecture 200 supportsNWDAF and management data analytics (MDAF) in a CSP application androbotic process automation (RPA). NWDAF may operate as application thatsupplements and compliments a variety of network functions as describedabove in FIG. 1B. The systems and methods described herein integratewith 5G data sources and with 3G and 4G applications. By way of exampleand not of limitation, the systems and methods described hereinintegrate with test equipment, a radio access network (RAN), thetransport layer and other such elements of a 3G, 4G and 5G communicationnetworks to create an end-to-end analytics framework that operatesacross different vendors and domains.

The systems and methods presented herein integrate with 5G and openradio access network (O-RAN) data sources. By way of example and not oflimitation, the integration is performed using an open API architecturethat operates using JavaScript open notation (JSON) to perform ahypertext transfer protocol (HTTP) request. The network analyticsarchitecture 200 collects data in a distributed environment and encryptsthe data near the point of collection. By way of example and not oflimitation, the systems and methods operate using a hybridplatform-as-a-service (PAAS) architecture, in which the core compute andstorage operations may be deployed in an illustrative Amazon WebServices (AWS) cloud and edge data collection and obfuscation can bedeployed on premises. Additionally, as described in further detailbelow, the systems and methods support robotic process automation (RPA)and integrate with legacy APIs or with complex process controls. Note,the RPA actions are monitored to validate that the RPA actions achievethe desired results.

The systems and methods overcome the issues associated with networkanalytics having non-standardized interfaces and inconsistent datacollection techniques by leveraging the network data analytics function(NWDAF), which is defined as part of the 5G architecture by 3GPP. NWDAFincorporates standard interfaces from the service-based architecture tocollect data by subscription or request from other network functions andsimilar procedures. These standard interfaces deliver analyticsfunctions in the network for automation or reporting, which overcomesthe challenges related to non-standardized interfaces and inconsistentdata collection.

The network analytics architecture 200 shown in FIGS. 2A, 2B and 2C usesan NWDAF events subscriptions service. The NWDAF subscription serviceenables network functions to subscribe to and unsubscribe from differentNWDAF analytic events. Also, the NWDAF subscription service notifies thenetwork functions about observed events. Observed events include loadlevel of network slice instance, service experience for an applicationor for a network slice, load analytics information for one or morenetwork functions, network performance in an area of interest, expectedbehaviour information for mobile devices, abnormal behaviour for mobiledevices, mobility information for mobile devices, communication patternsfor mobile device, congestion information in a specific location, andQoS sustainability. NWDAF provides analytics information for differentanalytic events to the consuming network functions. Additionally, NWDAFallows network functions to subscribe to and unsubscribe from periodicnotification and/or notification when an event is detected.

Referring to FIG. 2A there is shown analytics, services, maintenance,network management and network operations interfacing with NWDAF and acore compute and storage component. The network analytics architecture200 includes exemplar NWDAF use cases 202 that are listed by 3GPP for5G. The exemplar NWDAF use cases 202 include identifying a load-levelcomputation and prediction for a network slice instance. Another NWDAFuse case includes service experience computation and prediction for anapplication and/or user equipment (UE) group. Additionally, loadanalytics information and prediction for a specific network function(NF) may also be determined with NWDAF. Network load performancecomputation and future load prediction can also be performed with NWDAF.With respect to UE devices, NWDAF use cases include a UE expectedbehavior prediction, identifying abnormal behavior, anomaly detection,mobility related information, mobility related prediction, andcommunication pattern prediction. Another NWDAF use case relates tocongestion information at a current location and at a predicted locationand identifying or predicting changes to Quality of Service (QoS).

Although 3GPP has identified the various NWDAF use cases 202, theimplementation for the various use cases has not been provided by the3GPP. Additionally, the NWDAF use cases identified by 3GPP do not focuson developing analytics for ensuring the best customer experience in 5Gnetworks or hybrid networks that also include 4G capabilities. Thesystems and methods presented herein provide various implementations forQoE that measures the customer experience with an application or servicebased on the fulfilment of his or her expectation with respect to theapplication or service. The illustrative NWDAF use cases 202 interfacewith a core compute and storage component 204 that may be deployed in anAWS cloud, Azure cloud or as a platform-as-a-service (PaaS).

In the illustrative embodiment, the core compute and storage component204 is stored in a private or public cloud 206, which is accessed via anenterprise wide area network (WAN). Alternatively, the core compute andstorage component 204 may also be accessed via a data bus associatedwith a server, network appliance, or other such network component havinga data bus. Note the core compute and storage component 204 isassociated with the QoE network appliance 121 described above.

The core compute and storage component 204 also collects subscriber datawith the subscriber analytics module 210. The subscriber analyticsmodule 210 collects subscriber data such as financial data, geographicand demographic information, internet usage data and other such datathat can be used to generate customer profiles for determiningsubscriber QoE.

The core compute and storage component 204 also receives data related toIoT services 212. With respect to IoT services 212, IoT devices mustadhere to strict network usage and connect at a relatively low price.The strict limits of bandwidth, the number of session connections, theduration of the network connections and other such attributes arenecessary to establish the cost of supporting the IoT application. Fromthe CSP perspective, the IoT devices must be monitored to ensure theycomply with the agreed upon requirements. IoT devices are often highlydistributed and deployed in areas that are not easily serviced, thus,there is a need to easily resolve IoT issues efficiently. The systemsand methods described herein can be used to monitor IoT devices. Inoperation, an IoT baseline QoE may be established. By way of example andnot of limitation, the IoT baseline QoE may include a destination IPaddresses, IoT module type, e.g., vendor/model), connection frequency,and bandwidth consumed. Deviations from the IoT baseline QoE alert boththe CSP network and an IoT service provider.

The systems and methods described herein enable the CSP to comply withcontractually agreed parameters such as connectivity time, bandwidthconsumed, and other such network parameters. The IoT device issues maybe resolved by identifying the issues as a CSP network issue, an IoTservice provider issue or an IoT device manufacturer issue. By way ofexample and not of limitation, a provisioned IoT device that fails toconnect would be outside of normal behavior and additional data for rootcause analysis are reviewed. RAN logs are collected to determine if theIoT device had attempted to connect to the network and if CSP networkcongestion could be a reason the IoT has failed to connect. Thus,observing RAN logs would quickly determine if the IoT device wasattempting to connect or not. Also, it is possible to determine if theproblem is a one-off issue or a systemic issue. If the issue issystemic, the failure is quickly identified and provided to the IoTservice provider.

The core compute and storage component 204 also receives data frompredictive maintenance module 214. The predictive maintenance module 214measures historical and real-time data from network elements tounderstand the process of service degradation before failure.Additionally, the predictive maintenance module 214 predicts whichnetwork elements are more likely to fail in the upcoming days or hoursusing predictive analytics tools and techniques.

The core compute and storage component 204 is also communicativelycoupled with a network management module 216. The network managementmodule 216 performs network performance management, future provisioning,network vulnerability management and network energy infrastructuremanagement. Network management 212 is performed with remote monitoring,automated monitoring, troubleshooting, configuration, and optimizationof the network.

Furthermore, the core compute and storage component 204 iscommunicatively coupled to a network operations module 218. Networkoperations 218 refers to the activities performed by internal networkingstaff or third parties that the CSPs rely on to monitor, manage, andrespond to alerts on their network's availability and performance.

Referring to FIG. 2B there is shown the NWDAF edge collectorscommunicatively coupled to the core compute and storage component 204located in a private cloud or public cloud 206, which can be accessedwith a data bus or an enterprise wide area network (WAN). Atillustrative site A 220 there is a first site A NWDAF edge collector 222that interfaces with the 5G network functions (NF) 224 described above.Additionally, at site A 220 there is a second site A edge collector 226that interfaces with the open radio access network (O-RAN) distributedunit (DU) 228. As is well known in the art, the distributed unit (DU)sits close to the radio unit (RU) and runs the Radio Link Control,Medium Access Control, and parts of the Physical (PHY) layer. The RU isthe radio unit that handles the digital front end and the parts of thePHY layer and the digital beamforming functionality.

At illustrative site B 230 there is a site B NWDAF edge collector 232that interfaces with the open radio access network (O-RAN) centralizedunit (CU) 234. As is well known in the art, the CU runs the radioresource control (RRC) and packet data convergence protocol (PDCP)layers. The site B NWDAF edge collector 232 also interfaces with an openradio access network (O-RAN) RAN intelligent controller (RIC) 236. TheRIC 236 provides advanced control functionality, which deliversincreased efficiency and better radio resource management.

At illustrative site C 240 there is shown a first site C NWDAF edgecollector 242 that interfaces with subscribers' Wi-Fi access point(s)244. Also, at site C 240 there is a second site C edge collector 246that interfaces with data collection, analytics, and events (DCAE)associated with the Open Network Automation Platform (ONAP) 248. TheONAP platform enables product-independent capabilities for design,creation, and lifecycle management of network services. ONAP uniquelyprovides a unified operating framework for vendor-agnostic,policy-driven service design, implementation, analytics and lifecyclemanagement for large-scale workloads and services. The second site Cedge collector 246 may also interface with deep packet inspection (DPI)probes and other such inspection methods. DPI is used to locate, detect,categorize, block, or reroute packets that have specific code or datapayloads that are not detected, located, categorized, blocked, orredirected by conventional packet filtering. Unlike plain packetfiltering, deep packet inspection goes beyond examining packet headers.

For 5G architecture, 3GPP provides a series of logical interfaces forNWDAF and the management data analytics function (MDAF). In theillustrative embodiment, a variety of exemplary interfaces are used. Forexample, a first interface enables NWDAF to interact with theapplication function (AF) using the network layer service basedinterface (SBI). An illustrative second interface is associated with anN1 reference point, which is between the UE and the AMF, and the N2reference point that is between the RAN and the AMF. An illustrativethird interface includes the operational and management (O&M) layer,which configures the NF profile in the network repository function (NRF)and NWDAF collects the network function (NF) capacity information fromthe NRF. In an illustrative fourth interface, the MDAF interacts withapplication/tenant using northbound interfaces (NBI). In an illustrativefifth interface, the MDAF interacts with radio access network (RAN) dataanalytics function (DAF) using O&M layer service based interface (SBI).In an illustrative sixth interface, NWDAF consumes the services providedby MDAF using cross layer SBI. In an illustrative seventh interface,MDAF consumes the services provided by NWDAF using cross layer SBI. Inan illustrative eighth interface, MDAF collects data from NW layer viatrace file/monitoring services.

Referring to FIG. 2C there is shown a robotic process automation (RPA)module interfacing with the core compute and storage component. TheNWDAF data analytics described above are consumed by a robotic processautomation (RPA) module 270, which can provide actionable data and/orintelligence for planning systems, operations systems, and predictivemaintenance systems.

Generally, the RPA module 270 is used to manage and/or controloperational tasks by communicating actionable events to various networkfunctions or RAN systems that affect the QoE. In the illustrativeembodiment, the RPA module 270 receives the NWDAF data and identifiesactionable events, which may then be communicated to a service basedinterface (SBI) 272. The actionable events identified by the RPA module270 may also be used for network orchestration 274, RAN fine tuning 276,generating trouble tickets 278 and improving customer engagement 280.

The RPA module 270 may integrate with legacy APIs or complex processcontrols. In the illustrative embodiment, the RPA module is associatedwith the QoE network appliance, and where appropriate the impact of theRPA action is monitored to ensure the action achieved the desiredresults. Robotic process automation enables a CSP employee to configurea “robot” to emulate the actions of a human interacting within digitalsystems to execute a business process. RPA robots utilize the userinterface to capture data and manipulate applications just like humansdo. RPA robots interpret, trigger responses, and communicate with othersystems in order to perform a vast variety of repetitive tasks.

The systems and methods described herein provide an analytics strategythat includes an RPA module that receives and processes an integratedevent stream, which is described in FIG. 7 . The integrated event streamrefers to data captured by the WAN/data bus that is communicativelycoupled to the RPA module 270. For example, the integrated event streammay include the RAN data set, the CN data set, the NWDAF data set, theQoE latency measurement, the QoE bandwidth measurement, and the QoEpacket loss rate measurement. Additionally, the integrated event streammay include data sets collected from other edge based collectors.Furthermore, the integrated event stream may include network events. Asdescribed previously, network events may include all requests caused bya user interaction, a user action, a network interaction, and asubrequest. Note, not all network events are initiated by a user asdescribed previously.

In the illustrative embodiment, the integrated event stream is generatedby the QoE network appliance. The integrated event stream provides atemporal perspective, a geographic perspective, and a topologicalperspective. These different perspectives can be used to correlateevents and identify degradation of service and further identify the rootissue that is causing the problem. The integrated event stream providesmore valuable information than a single NWDAF event stream becausecorrelations can be made at various wireless communication levels, fromanalyzing the various data sets and network events to determine thecause of the network degradation.

The RPA module receives the integrated event stream and can separate theintegrated event stream data into a temporal event stream, a geographicevent stream and a topological event stream. Each event stream isprocessed by the RPA module 270, which is communicatively coupled to oneor more automation modules that perform the actionable events. Theillustrative automation modules may include the application servicebased interface (SBI) module 272, a network orchestration module 274, aRAN fine tuning module 276, a trouble tickets module 278 and a customerengagement module 280.

Referring to FIG. 3A and FIG. 3B there is shown an illustrativehigh-level network analytic billing system that charges subscribersbased on the QoE. FIG. 1 presented an OPS-IQ software module thatintegrated data sets from 4G networks, 5G networks and hybrid networks,and then the integrated data sets were communicated using NWDAF. FIGS.2A, 2B, 2C presented a system that used NWDAF to obtain real-time orpseudo real-time network analytics for data analysis and to determineactions that affect various QoE parameters. FIG. 3A presents theillustrative network analytic billing system 300 for billing subscribersbased on the QoE that is determined using the NWDAF data sets.

The illustrative network analytic billing system 300 receives QoE datafrom deep packet inspection (DPI) and event detail records (EDR) module302. As previously described, DPI is used to locate, detect, categorize,block, or reroute packets that have specific code or data payloads thatare not detected, located, categorized, blocked, or redirected byconventional packet filtering. EDR refers to event characteristics forany activity that can be monitored.

Additionally, the network analytic billing system 300 receives data fromvarious radio access networks (RAN) 304 such as legacy RAN, O-RAN DU,O-RAN CU and O-RAN RIC. Legacy RAN platforms are based on proprietaryhardware. The open radio access network (O-RAN) distributed unit (DU)sits close to the radio unit (RU) and runs the RLC, MAC, and parts ofthe PHY layer. The RU is the radio unit that handles the digital frontend and the parts of the PHY layer and the digital beamformingfunctionality. The open radio access network (O-RAN) centralized unit(CU) 234 runs the radio resource control (RRC) and packet dataconvergence protocol (PDCP) layers. The open radio access network(O-RAN) RAN intelligent controller (RIC) 236 provides advanced controlfunctionality, which delivers increased efficiency and better radioresource management.

The network analytic billing system also receives data from networkfunctions 306 such as the network slice selection function (NSSF) whichsupports taking information from the NWDAF into consideration for slideselection. The NSSF accesses the NWDAF subscription service. The accessand mobility management (AMF) function includes registration management,connection management, reachability management, mobility management andvarious functions related to security and access management andauthorization. With respect to the NWDAF interaction and the AMFfunction, AMF subscribes to NWDAF to take session management function(SMF) load information from NWDAF into consideration for SMF selection.The Policy Control Function (PCF) supports the unified policy frameworkthat governs network behavior. The PCF provides policy rules to controlplane functions to enforce them. The NWDAF interaction takes input fromthe NWDAF into consideration for policies on assignment of networkresources and for traffic steering policies.

The Network Exposure Function (NEF) provides a way to securely exposethe services and capabilities provided by a 3GPP network functions,e.g., third party, internal exposure, and internal re-exposure. TheNWDAF interaction supports forwarding UE mobility information from NWDAFto the Application Function (AF) when it is untrusted. Additionally, theNWDAF interaction also supports forwarding UE communication informationfrom NWDAF to the AF when it is untrusted. Furthermore, the NWDAFinteraction forwards user data congestion information from NWDAF to theAF when it is trusted.

The DPI data 302, RAN data 304 and NWDAF data 306 are received by anillustrative network analytics cloud component 308 that may operatesimilarly to core compute and storage component 204 described above andshown in FIGS. 2A-C. Other data received by the network analytic cloudcomponent 308 includes, by way of example and not of limitation,planning reports 310, automated inputs to self-organizing networks (SON)312, failure prediction 314, scale up/down signaling 316, performanceprediction 318, root cause analysis 320, content usage analysis andforecasting 322, per subscriber service level agreement (SLA) 324, persubscriber usage forecasting 326, per subscriber resource utilization328.

More specifically, planning reports 310 include at least one ofstatistical information about infrastructure, expenditures, use, datafor optimizing infrastructure and identify ways to reduce costs.Automated inputs for self-organizing networks (SON) 312 refers to moredense and complex networks having automated inputs that are associatedwith self-organizing networks (SON). Failure prediction 314 refers topredictive analysis that predicts network failures. Scale-up/downsignaling 316 enables CSPs to conduct infrastructure capacity planning,select and deploy resources in the cloud or at the edge. Performanceprediction 318 refers to using existing measurement data to predictnetwork performance where direct measurements are not available. Rootcause analysis 320 refers to problem solving that is used to identifythe root causes of faults or problems. Content usage analysis andforecasting 322 relates to the analysis of the content usage andforecasting content usage. Per subscriber service level agreement (SLA)324 refers to the CSP monitoring subscriber services agreed upon betweenthe subscriber and the CSP. Per subscriber usage forecasting 326 refersto forecasting the usage on a per subscriber basis. Per subscriberresource utilization 328 refers to the utilization of the network on aper subscriber basis.

The network analytic cloud component 308 passes the various data sets tothe billing analytics module 330, which is depicted in further detail inFIG. 3B. The billing analytics module 330 includes a usage data platform332. By way of example and not of limitation, an illustrative usage dataplatform 332 includes Digital Route. The illustrative usage dataplatform 332 converts raw usage data into billable items in order toenable usage based business models. The platform interacts with the mostcritical revenue systems, ensuring accurate billing to subscribers.

An operations support system (OSS) 334 is communicatively coupled to theusage data platform 332. OSS 334 includes order management, networkinventory management and network operations.

A subscriber database (DB) 336 is also communicatively coupled to theusage data platform 332. The subscriber DB 336 includes a database withsubscriber information such as billing address and personal informationabout the subscriber. The subscriber DB 336 may also include metadataabout mobile applications that the subscriber opens and the durationthat the mobile application is open.

The call data record (CDR) 338 or extended data record interfaces withthe usage data platform 332. Note, for purposes of this patent the calldata record 338 also refers to an extended data record.

Network analytics provide true visibility into each subscriber session.Thus, if the service is delivered as expected, the call detail record(CDR) 338 would reflect that the service was delivered as expected. Ifthe service is not delivered as expected, the CDR would also reflectthat the service was delivered inadequately. This ability to monitor thesubscriber session makes it possible for the CSP to bill the subscriberbased on each subscribers QoE. In operation, the hybrid 5G and 5Gnetwork analytics tabulate all the resources utilized, verify theservice the subscriber was consuming, verify that the network deliveredthose packets with the quality expected by the customer, and generate aCDR that is passed to billing.

The alarms 340 are communicatively coupled to the usage data platform332. The alarms alert the CSP to problems indicated by the CSP. Theseproblems are weighted based on severity levels as critical, major, orminor.

The online charging system (OCS) 342 also interfaces with the usage dataplatform. The OCS is a system allowing a CSP to charge theirsubscribers, in real time, based on their service usage.

A fraud management component 344 is communicatively coupled to the usagedata platform 332. The fraud management component 344 monitors calls inreal time for suspicious traffic or call patterns.

A postpaid billing 346 interfaces with the usage data platform 332.Postpaid billing 346 bills the subscriber after the service has beenprovided based on the use of the mobile service. Generally, the mobileservices are billed at the end of each month.

The systems and methods described herein enable the CSP to bill thesubscriber based on the quality of experience (QoE). By way of exampleand not of limitation, a subscriber and CSP agree to particular type ofQoE video quality, and the subscriber is charged on a monthly basis forthe QoE video quality. However, if the CSP does not provide the QoEvideo quality, the subscriber is not charged for the QoE video qualityservice.

Referring to FIG. 4 there is shown a block diagram of various componentsfor an event based billing mediation system 400, which operates with thesystems described in FIG. 1 through FIGS. 3A and 3B. In summary, thereare two separate processes occurring in FIG. 4 . The first process 402is a real-time event analysis, which determines the quality ofexperience (QoE) based on the service level agreement (SLA) 324. Thesecond process 404 collects and normalizes the data with the usage dataplatform 332 for event based billing. The core switch interface (CSIF)call detail record (CDR) 406 having the QoE SLA data is passed to theusage data platform.

The event based billing mediation system 400 can operate for variousnetwork slices and can support different business models such asdifferent business-to-business applications (B2B), e.g., IoTapplications and subscriber applications. The different B2B applicationhave different service level agreement (SLA) and QoE, which are based onvarious network performance variables. By way of example and not oflimitation, the different network performance variables include alatency variable, a throughput variable, a maximum utilized bandwidthvariable, a maximum packet loss rate variable, a service experiencestatistics variable, a number of protocol data unit (PDU) sessionvariables, a registered subscriber variable, a load level informationvariable, and other such variables. In 5G, the protocol data unit (PDU)session provides end-to-end user plane connectivity between the mobiledevice and a specific data network through the user plane function. APDU session supports one or more QoS flows.

In operation, the illustrative event billing mediation system 400receives a plurality of analytics functions 408 which are processed asnetwork analytics using NWDAF 410, which notifies the core switchinterface (CSIF) 412. The CSIF then transmits a charging data request toa charging function (CHF) 414. The CHF 414 can include both an onlinecharging function and a charging data function. The CHF then passes acall detail record (CDR) to a charging gateway function (CGF) 416. TheCGF 416 acts as a gateway between the CSP network and the billing domain(not shown).

The CSP network analytics described above provide the CSP with anunderstanding of the cost of providing the subscriber service(s). Thesystems described above utilize network analytics to charge differentlyfor different QoE. In operation, the hybrid 5G and 5G network analyticstabulate all the resources utilized, verify the service the subscriberwas consuming, verify that the network delivered those packets with thequality expected by the customer, and generate a detailed Call DetailRecord (CDR) that is passed to billing as described above.

Depending on whether the CSP network satisfied the customer QoE, the CSPbills the subscriber based on delivering packets with the qualityexpected and the customer would pay for the improved experience. If theQoE was not satisfied, the subscriber would not be billed and the CSPwould have a full record of the reason the network did not meet the QoErequirements.

Referring to FIGS. 5A, 5B and 5C there is shown various methods forcommunicating application QoE requirements and the integration of theQoE requirements with applications. An “application” includes a mobileapplication downloaded from an app store or pre-loaded on a computingdevice or upgrading a mobile application to provide new services.Subsequently, the application is executed on the mobile device and iscommunicatively coupled to the Internet using the CSP network. Theillustrative application operates according to the terms of a servicelevel agreement (SLA), which is a commitment between the CSP and thesubscriber that relates to the expected QoE. Although, the subscriber istypically an individual, the subscriber may also be an IoT device, andIoT service or a mobile virtual network operation (MVNO).

In a first embodiment, the application is registered with the CSP.During this registration process the CSP and the application ownercreate an authentication code that is used to validate the authenticityof the application owner. For example, the authentication code mayinclude the application name, which is registered with the CSP network.The CSP network associates the authentication code with a QoErequirement that includes a minimum bandwidth. Other QoE requirementsinclude latency, jitter, and packet loss requirements. In operation, theinformation passed from the application to the CSP include one or moreof an authentication code, an IP Address for the subscriber, an IMSI orMSISDN, an IP address of the server, a name of the application, anapplication start time and other such application information.Additionally, during the application session, QoE parameters aremonitored and used to optimize CSP network performance as describedherein.

In the embodiment shown in FIG. 5A, the application is downloaded fromthe app store at arrow 502. Also, a message is sent from the mobiledevice to the CSP, at arrow 504, with the name of the application, thesubscriber ID (IMSI, MSISDN), and the bandwidth requirements, packetloss rate needs, and latency to the application QoE server. Theapplication QoE server is associated with the CSP network and may alsobe referred to as a QoE network appliance as described herein. The term“QoE server” is provided for simplicity.

In the illustrative embodiment shown in FIG. 5B, the CSP detects thatthe application is on the CSP network after the subscriber downloads theapplication, at arrow 512. When the application is launched, a messageis sent to a designated QoE server in the CSP network at arrow 514. Themessage sent at arrow 514 makes the CSP network aware that theapplication is running. When the application is terminated, theapplication sends a close message to the server.

In the illustrative embodiment shown in FIG. 5C, the applicationoperates in a manner similar to that of FIG. 5B, except there areadditional process steps that relate to QoE updates due to changingapplication requirements. Therefore, process steps associated with arrow512 and arrow 514 continue as in FIG. 5B. Additionally, there is aprocess step 516, in which the application sends a message to theapplication QoE server to provide mobile application updates and toimprove the QoE requirements from interim or preliminary QoErequirements—that were obtained during the initial download. Forexample, the application may be running in the background and consumingfew network resources, and when the application is in use by thesubscriber the QoE requirements are updated accordingly. In anotherillustrative embodiment, a gaming application may require more networkresources to satisfy the QoE during game play—so the update alerts theCSP of the need for more network resources.

In a fifth embodiment (not shown), the CSP network would observe theintegrated event stream to identify that the application is running. TheCSP network then proceeds to generate a time-based baseline of CSPnetwork resources needed to support the application. The time-basedbaseline is used to determine the QoE requirements for the application.Additionally, the CSP network can monitor the measured QoE over anextended period of time. The CSP determines the QoE requirements for theapplication with inferential systems, such as DPI, NWDAF networkanalytics, or other such analytics.

The network analytics used to determine the QoE requirement are notlimited to bandwidth, latency, and packet loss. Other network analyticsmay also be used to determine the QoE requirement including CNanalytics, NWDAF analytics, RAN analytics, DPI analytics, and contentanalytics that continuously monitor the network efficiency, subscriberusage, and service operational patterns.

In operation, edge collection operates by collecting data from anetworked device operating at the “edge” of the network. For example,NWDAF may perform edge collection as described above. Also, DPI isgathered using an edge collection process and/or module. Furthermore,RAN data can also be collected, but the collection process is slower.RAN data is also collected on the edge of the network. The edgecollection process communicates the received analytic data sets to thecore network QoE network appliance. The analytic data sets can becollected from a single location or from multiple locations. By way ofexample and not of limitation, the illustrative core network is GUAVUS®and/or THALES® compute and storage network component, which is alsoreferred to as a QoE network appliance.

The collected data sets may be stored on the edge of the network or atthe QoE network appliance. The analytic data sets can vary and depend onthe particular type of traffic that is gathered at the edge location.The core compute and storage associated with the QoE network appliancesupport use cases across the whole network and gathers data from aplurality of edge locations.

In general, the data sets collected by the core compute system is usedto generate an “action.” The action may have an associated API that canbe used to provide process automation or access to other APIs. APIs areused to process data sets, generate actions, and communicate with otherAPIs. By way of example and not of limitation, the illustrative systemsand methods teach that edge collection of data in a distributed networkthat is received and processed by a core compute and storage networkcomponent 204 that generates an integrated event stream which canundergo a robotic process automation (RPA) and/or be communicated to abilling network component. The network analytics architecture supportedby the systems and methods described herein is highly distributed and isnot standards based.

The network analytics continuously provide detailed end-to-endvisibility from the RAN to the Internet to ensure resources are runningoptimally for the entire subscriber base. The systems and methodsdescribed herein monitor resource utilization, content demand forecasts,and operational analytics to give the CSP visibility and control thatallows additional services to be operated. Additionally, continuouscontent analytics ensure that the work-from-home slice is being used inaccordance with CSP policies to identify fraudulent usage.

The systems and methods combine network analytics and billing, whichgives the CSP the ability to offer new services and monetize the newservices. The systems and methods enable CSPs to offer new service levelagreements (SLAs) to subscribers and IoT devices, which represents a newrevenue model for CSP networks.

Referring to FIGS. 6A, 6B, and 6C there is shown an illustrativeflowchart for determining a QoE requirement, measuring satisfaction ofQoE, billing for QoE, resolving unsatisfactory QoE. More specifically,the method 600 satisfies a quality of experience (QoE) requirement foran application executed on a mobile device that accesses a CSP network.Additionally, the method 600 determines a CSP network policy based onthe QoE. Furthermore, the method 600 resolves an unsatisfactory QoE foran application executed on a wireless device that accesses acommunication service provider (CSP) network.

In FIG. 6A, the method 600 is initiated at block 602 where anillustrative mobile application is registered for QoE. By way of exampleand not of limitation, the mobile application is downloaded from anapplication store, which is also referred to as an “app store.” In oneembodiment, downloading of the mobile application from the app store iscommunicated to one or more network appliances associated with the CSPnetwork.

At block 604, an illustrative authentication code is communicated to theQoE network appliance. The method also includes registering anauthentication code, which is associated with the mobile application.The illustrative authentication code is used to authenticate the UEcommunications with the CSP network.

In the illustrative embodiment, the method also registers an applicationname associated with the mobile application. The method proceeds tocommunicate at least one of the authentication code and the applicationname from the mobile application, which is executed on the UE, to theQoE network appliance. In a further illustrative embodiment, the RANdata set and an authentication data set determine when a wireless deviceis communicatively coupled to the CSP network.

At block 606, the RAN data, CN data, and NWDAF data related to thesubscriber QoE are identified and/or communicated to the QoE networkappliance. Additionally, an IP address associated with the subscriber, asubscriber ID, an IP address for a QoE network appliance, and anapplication start time may be identified and/or communicated to the QoEnetwork appliance.

In the illustrative embodiment, an edge-based collection module gathersthe RAN data set, the CN data set and the NWDAF data set. Additionally,the edge-based collection module may be configured to reduce a volume ofdata from the RAN data set, the CN data set and NWDAF data set beforecommunicating the reduced volume of data to the QoE network appliance.

In a further illustrative embodiment, a deep packet inspection (DPI)data set is also identified and/or communicated to the networkappliance. The DPI data set is selected from the open systeminterconnection (OSI) group consisting of a network layer, a transportlayer, a session layer, a presentation layer, and an application layer.In the illustrative embodiment, the DPI data set is also associated withthe latency measurement, the bandwidth measurement, and the packet lossrate measurement.

The RAN data set, the CN data set and the NWDAF data set are associatedwith a latency requirement, a bandwidth requirement, and a packet rateloss requirement to determine the subscriber QoE requirement for themobile application. Additionally, the systems described above may useedge processing for improved latency response by optimizing networkresources with data sets that include one or more of RAN data, CN data,NWDAF data, DPI data and other such data that can be associated with alatency response.

At block 608, the subscriber QoE requirement is determined aftergathering the RAN data set, the CN data set, the NWDAF data set with theedge-based collection module. In an illustrative embodiment, thesubscriber QoE requirement is determined at the QoE network appliancewith at least one of the RAN data set, the CN data set and the NWDAFdata set.

In one illustrative embodiment, the method for determining a QoErequirement includes registering the QoE requirement that is associatedwith the mobile application and the QoE requirement is communicated fromthe mobile application executed on the UE to the QoE network appliance.

In another embodiment, the method for determining a QoE requirementincludes communicating a QoE message that is sent from the UE to the CSPnetwork after downloading the mobile application to the mobile device.The QoE message includes the QoE requirement for the illustrative mobileapplication and the name of the application.

In yet another embodiment, the method for determining a QoE requirementincludes launching the mobile application on the UE and communicating aQoE message from the mobile device to the QoE network appliance. The QoEmessage indicates that the mobile application is being executed on themobile device and a “close” message is communicated from the UE to theQoE network appliance when the mobile application is terminated.

In a further embodiment, the method for determining a QoE requirementincludes initiating the mobile application and generating an updated QoEmessage that is communicated from the mobile application to the QoEnetwork appliance, while the mobile application is being executed on themobile device. The QoE message includes an updated activity for themobile application and an interim QoE requirement corresponding to theupdated activity. The CSP network then proceeds to modify the networkresources to satisfy the interim QoE requirement.

In still a further embodiment, the method for determining a QoErequirement further includes enabling the CSP network to generate aninferred mobile application QoE requirement based on the mobileapplication usage of CSP network resources. The inferred mobileapplication QoE requirements includes an inferred latency requirement,an inferred bandwidth requirement and an inferred packet loss raterequirement.

In yet another illustrative embodiment, determining the application QoErequirement includes generating a QoE fingerprint that can be used toidentify the mobile application. Additionally, the QoE fingerprintincludes the RAN data set, the CN data set and NWDAF data set.

At decision diamond 610, data sets are gathered during an illustrativecall session. For the illustrative DPI data set, the data may begathered over a time interval such as two to three minutes and then theDPI data set is passed to the block 614 to determine the measured QoEscore. For the illustrative RAN data, CN data and NWDAF data, the datasets are gathered in real-time at block 612 and passed to block 614.

At block 614, a measured QoE score is generated for the subscribersession with the illustrative mobile application. The measured QoE scoreis determined in real-time. The measured Quality of Experience (QoE)score is generated with the RAN data set, the NWDAF data set and the CNdata set. Additionally, the measured QoE score is associated with thelatency measurement, the bandwidth measurement, and the packet loss ratemeasurement.

In another embodiment, the method includes determining, with the DPIdata set, an optimal bandwidth measurement for a particular application.The optimal bandwidth measurement is associated with the measured QoEscore. Additionally, the DPI data set can be used to determine anoptimal latency measurement for the particular application, in which theoptimal latency measurement is associated with the measured QoE score.The DPI data set may also be used to determine when the measured QoEscore satisfies the customer QoE requirement.

In a further embodiment, the method associates a QoE data record with aQoE API, in which the QoE data record includes the measured QoE score.Also, a notification is generated about changes to the measured QoE withthe QoE API.

The method 600 continues in FIG. 6B at block 616, where the measured QoEscore is compared to the QoE requirement. In one illustrativeembodiment, the process step at block 616 determines when the measuredQoE score satisfies the customer QoE requirement by comparing thelatency requirement, the bandwidth requirement, and the packet loss raterequirement associated with the customer QoE requirement, with thelatency measurement, the bandwidth measurement, and the packet loss ratemeasurement associated with the measured QoE score.

In another embodiment, the process step 616 communicates a QoE messagethat is sent from the mobile device to the CSP network after downloadingthe mobile application to the mobile device. The QoE message includesthe application QoE requirement for the application. The CSP networkdetects the mobile application on the CSP network. The measured QoEscore is monitored for compliance with the application QoE requirement.The measured QoE score is stored so that other mobile devices using themobile application can access the RAN data set, the NWDAF data set andthe CN data set.

In yet another embodiment, the process step 616 includes forecasting, atthe QoE network appliance, a per device network load for a scaling modelwith the RAN data set, the CN data set and the NWDAF data set. In yetanother embodiment, the method includes forecasting, at the networkappliance, a per application network load for a scaling model with theRAN data set, the CN data set and the NWDAF data set. In still anotherembodiment, the method includes forecasting, at the network appliance, aper location network load for a scaling model with the RAN data set, theCN data set and the NWDAF data set.

At decision diamond 618, the network policy is continuously reevaluatedto determine whether a change in CSP network policy is needed.Initially, the CSP network policy is based on the QoE requirement. Also,the measured QoE score may be used to establish the network policy. Themeasured QoE score is shared as a call data record that is associatedwith a particular subscriber ID. In the illustrative embodiment, abilling system records the QoE requirement associated with the mobileapplication, the measured QoE score, and the CSP network policy in acall data record.

A change in the network policy may be based on the measured QoE scorefor a particular subscriber at a particular location, which isassociated with a particular cell. When reduced network performance isdetected with a low measured QoE score at an impacted area, the CSPnetwork alerts the UE of the reduced network performance. The reducednetwork performance can also be used to determine how to charge the userfor services as described herein.

By way of example and not of limitation, the change in network mayresult in increasing the pricing to maintain the QoE or reducing thepricing due to the low measured QoE score. If there is a change innetwork policy, the change may result in changing the QoE requirement(not shown) or continuing to measure the QoE score to determine if thereis a change in measured QoE score at block 614.

At block 620, the particular subscriber is billed for services when themeasured QoE score satisfies QoE requirement for the particular callsession. In operation, the subscriber ID is charged when the measuredQoE score satisfies the QoE requirement.

At decision diamond 622, a determination is made whether to change theQoE requirement or change the pricing based on measured QoE score. Theillustrative QoE network appliance determines how to charge theparticular subscriber based on the initial QoE requirement, the modifiedQoE requirement, and the measured QoE score during each call session.When the method 600 determines to change the QoE requirement or pricing,the methods 600 returns to block 616. When the method 600 determines notto change the QoE requirement or pricing, the methods 600 proceeds todecision diamond 624.

At decision diamond 624, a determination is made regarding a callsession having a consistently low measured QoE score. If the measuredQoE score during the call session is satisfactory, the method ends.However, if the call session(s) have a consistently low measured QoEscore, the decision diamond may indicate that an unsatisfactory QoE isnot caused by the CSP network because the CSP network satisfies theapplication QoE requirement.

In operation, the decision diamond 624 may have a correlation modulethat determines that a low measured QoE score affects a plurality ofwireless devices. For example, faulty WiFi associated with thecustomer's premises may be the cause of a low measured QoE score; andthe correlation module detects the faulty Wi-Fi.

If a consistently low measured QoE score is identified at decisiondiamond 624, the method proceeds to block 626. At block 626, a networkorchestration module adds or removes network resources. In operation, anillustration QoE message is communicated to the network orchestrationmodule, which then adds or removes one or more network resources, whichchanges the measured QoE score.

If the consistently low measured QoE score does not improve, the methodproceeds to block 628 in FIG. 6C where an anomaly detection step isperformed. An “anomaly” is detected within the illustrative RAN dataset, the CN data set, and the NWDAF data set by identifying a behaviorassociated with the wireless device that causes the measured QoE scoreto not satisfy the application QoE requirement when the CSP networkshould satisfy the application QoE requirement. The anomaly detectionstep may include generating a baseline based on location to determineCSP network resource requirements for each mobile application. Once thebaseline is determined, an effective operational range is associatedwith the baseline, and any data points that fall outside of theoperational range are identified as anomalies.

At block 630, a root cause analysis (RCA) is performed when the stepsdescribed in blocks 626 and 628 fail. Root cause analysis refers to aprocess of problem solving to determine the root causes of faults andproblems.

At block 632, a congestion notification is generated and communicated.There are two types of congestion notifications, namely, a networkcongestion notification and subscriber congestion notification.

Network congestion is congestion that is detected on a RAN or slice. Anetwork congestion notification is generated and communicated to billinganalytics 330 (shown in FIG. 3 a ) and other back office systems asneeded. The purpose for the network congestion notification is to enablethe CSP to take particular actions, which may include alerting thesubscriber of network congestion, trigger increased billing rates,trigger changes in network policy to restrict network resources madeavailable to one or more subscribers, trigger an increase in networkresources by expanding cloud-based computing resources. By way ofexample and not of limitation, the network congestion notification alsoincludes time, subscriber ID(s), top applications in use by subscribersordered by bandwidth consumed, slice ID(s), cell ID(s), congestiondegree (1 to 10 with 1 being least and 10 being most), and congestionprediction which predicts the duration of a congestion period inminutes.

A subscriber congestion notification would be sent from NetworkAnalytics to one or more back office systems to provide a full record ofthe impact of the congestion to the subscriber. Similar to a networkcongestion notification, the subscriber level congestion notificationprovides per subscriber level of information to the CSP with detailedinformation of network usage during the congestion event. By way ofexample and not of limitation, the congestion notification includessubscriber ID, applications in use by subscribers and ordered bybandwidth consumed, slice ID, cell ID(s), congestion degree from 1 to 10with 1 being least and 10 being most, congestion prediction withpredicted duration of congestion period in minutes, RAN latency beforean event and during the event, RAN throughput before the event andduring the event, and maximum usable bandwidth before the event andduring the event.

Referring to FIG. 7 there is shown a method 700 for generating anintegrated event stream that is communicated to a robotic processautomation (RPA) module. The method 700 begins at block 702 where theRAN data set, the CN data set and the NWDAF data set are associated witha QoE requirement that includes, but is not limited to, a QoE latencyrequirement, a QoE bandwidth requirement, and a QoE packet rate lossrequirement as described above.

Note, the systems described above may use edge processing for improvedlatency response by optimizing network resources with data sets thatinclude one or more of RAN data, CN data, NWDAF data, DPI data and othersuch data that can be associated with a latency response.

The illustrative process step at block 702 monitors data sets at aparticular edge-based collection module, e.g., an eNodeB base station.Also, the process step at block 702 monitors the data sets at aplurality of edge-based locations for different types of networktraffic. Note, the systems and methods collect data from the edge of thenetwork and process the data in a distributed manner.

At block 704, the edge-based collection module(s) then pass the data tothe QoE network appliance that includes a core compute component and astorage component. The QoE network appliance then stitches the datatogether to generate a consolidated call data record. In other words,the data from one or more edge-based collection modules is “stitched”together at the QoE network appliance with data sets collected orgathered from the edge-based collectors.

There may be different types of stitched data based on differentservices. For example, in a fiber network, the stitched data comes fromedge-based collectors associated with the fiber network. An IP networkhas different network devices than the fiber network and, consequently,the service level will be different for the IP network versus the fibernetwork. In the illustrative embodiment, the data that is stitched orcombined together includes, but is not limited to, RAN data sets, CNdata sets, NWDAF data sets and DPI data sets.

At block 706, an integrated event stream is then generated. Theintegrated event stream includes the RAN data set, the CN data set, theNWDAF data set, the QoE latency measurement, the QoE bandwidthmeasurement, and the QoE packet loss rate measurement. Additionally, theintegrated event stream includes data sets collected from otheredge-based collectors. Also, the integrated event stream may includenetwork events. As described previously, network events may include allrequests caused by a user interaction, a user action, a networkinteraction, and a subrequest. Note, not all network events areinitiated by a user as described previously.

The integrated event stream is generated by the QoE network appliance.The integrated event stream provides a temporal perspective, ageographic perspective, and a topological perspective. These differentperspectives can be used to correlate events and identify degradation ofservice and further identify the root issue that is causing the problem.

For example, if there is an outage in the fiber network that has rippleeffects across the IP network, then the outage in the fiber network mayor may not service degradation to the customer at the IP network level.The systems and methods described herein enable a CSP network toevaluate all the events in the network to identify patterns thatindicate there is actually a service degradation to the subscriber QoE.If service degradation is detected at the QoE level, then the systemsand methods described herein can be used to identify “what” happened, atthe “what” layer to cause the service degradation to occur.

The integrated event stream provides more valuable information than asingle NWDAF event stream because correlations can be made at variouswireless communication levels, from analyzing the various data sets andnetwork events to determine the cause of the network degradation. Morespecifically, correlations can be based on time, geography, and networktopology. Note, network topology refers to all areas in the network thatare having issues other than just a particular geographical location.Thus, the system and method described herein is performing a temporalcorrelation, a geographical correlation, and a topological correlation.

The integrated event stream is then transferred to the robotics processautomation (RPA) module or a custom API integration. Additionally, theintegrated event stream may be communicated to a central repository thatcan be used to generate the measured QoE score.

In one illustrative embodiment, the integrated event stream is a networkanalytic that is communicated as a “data record.” By way of example andnot of limitation, the data record is generated at the end of a TCPsession with cumulative statistics related to the session. Interim datarecords may be generated before the end of a TCP session. Data recordsmay be produced in the event of congestion or could be generated for allsessions depending on the needs of the CSP. The data records may includestart/stop time, unique record ID, subscriber ID (IP Address, IMSI,MSISDN), application ID(s), RAN latency, RAN throughput, maximumutilized, bandwidth, maximum packet loss rate, service experiencestatistics, number of PDU sessions, slice load level information andmeasured QoE score.

At block 708, the integrated event stream is used as an analytic inputfor the robotics process automation (RPA) module. The systems andmethods described above provide an architecture that supports automatedactions being performed by the RPA. The RPA includes a set of testcriteria for the action taken, in which the test criteria determine thesuccess or failure for the action taken.

Additionally, a remote action engine (RAE) associates a result with theaction taken by the RPA. The RAE then proceeds to test whether theaction taken produced the desired result based on a test criteria. Thetest criteria is associated with the action taken to determine if theaction taken produced the desired result.

For example, the process steps described may be applied to anillustrative voice call to determine the QoE of the voice call.Previously, the data for related to setting up a call, determining thequality of the call, and analysis of call termination were analyzedseparately—so there was no integrated data stream. The systems andmethods described herein bring this disparate data together as anintegrated data stream that is used as a single index to analyze callquality by determining the measured QoE score.

It is to be understood that the detailed description of illustrativeembodiments is provided for illustrative purposes. The scope of theclaims is not limited to these specific embodiments or examples.Therefore, various process limitations, elements, details, and uses candiffer from those just described, or be expanded on or implemented usingtechnologies not yet commercially viable, and yet still be within theinventive concepts of the present disclosure. The scope of the inventionis determined by the following claims and their legal equivalents.

What is claimed is:
 1. A method for measuring quality of experience(QoE) satisfaction for an application associated with one or moresubscribers that are accessing a CSP network, the method comprising:executing the application on a mobile device that is communicativelycoupled to the CSP network; identifying a quality of experience (QoE)requirement for the application associated with one or more subscribersaccessing the CSP network, in which the QoE requirement for theapplication includes a QoE latency requirement, a QoE bandwidthrequirement, and a QoE packet loss rate requirement; identifying a radioaccess network (RAN) data set; identifying a core network (CN) data setthat includes a network data analytics function (NWDAF) data set;gathering the RAN data set, the CN data set and the NWDAF data set withan edge-collection module that communicates the RAN data set, the CNdata set and the NWDAF data set to a QoE network appliance, whichincludes a core compute and storage network component; associating, atthe QoE network appliance, the RAN data set, the CN data set, and theNWDAF data set with a QoE latency measurement, a QoE bandwidthmeasurement, and a QoE packet loss rate measurement; determining, at theQoE network appliance, the QoE requirement with the RAN data set, the CNdata set, and the NWDAF data set; generating, at the QoE networkappliance, a measured QoE score with the RAN data set, the CN data set,and the NWDAF data set wherein the measured QoE score is associated withthe latency measurement, the bandwidth measurement, and the packet lossrate measurement; and determining when the measured QoE score satisfiesthe QoE requirement by comparing the latency requirement the bandwidthrequirement, and the packet loss rate requirement associated with theQoE requirement, with the latency measurement, the bandwidthmeasurement, and the packet loss rate measurement associated with themeasured QoE score.
 2. The method of claim 1 further comprisingidentifying a deep packet inspection (DPI) data set, wherein the DPIdata set is selected from the open system interconnection (OSI) groupconsisting of a network layer, a transport layer, a session layer, apresentation layer, and an application layer; and wherein the DPI dataset is associated with the latency measurement, the bandwidthmeasurement, and the packet loss rate measurement; wherein the DPI dataset is also used to determine when the measured QoE score satisfies theQoE requirement.
 3. The method of claim 2 wherein edge-based collectionmodule reduces a volume of data from at least one of the DPI data set,the RAN data set, the NWDAF data and the CN data set beforecommunicating the reduced volume of data to the QoE network appliance.4. The method of claim 2 further comprising determining, with the DPIdata set, an optimal bandwidth measurement for a particular application,in which the optimal bandwidth measurement is associated with themeasured QoE score.
 5. The method of claim 2 further comprisingdetermining, with the DPI data set, an optimal latency measurement forthe particular application, in which the optimal latency measurement isassociated with the measured QoE score.
 6. The method of claim 1 furthercomprising forecasting, at the QoE network appliance, a per devicenetwork load for a scaling model with at least one of the RAN data set,the NWDAF data set and the CN data set.
 7. The method of claim 1 furthercomprising forecasting, at the QoE network appliance, a per applicationnetwork load for a scaling model with the DPI data set, the RAN dataset, the NWDAF data set and the CN data set.
 8. The method of claim 1further comprising forecasting, at the QoE network appliance, a perlocation network load for a scaling model with the DPI data set, the RANdata set, the NWDAF data set and the CN data set.
 9. The method of claim1 further comprising determining a CSP network policy; determining areduced network performance with the measured QoE score at an impactedarea; and enabling the CSP network to alert at least one mobile deviceof the reduced network performance.
 10. The method of claim 1 furthercomprising determining a CSP network policy; and changing the CSPnetwork policy based on the measured QoE for the subscriber ID at aparticular cell.
 11. The method of claim 1 further comprising billing asubscriber ID when a charging function determines the measured QoEsatisfies the QoE requirement; and having the charging function, whichis communicatively couple to a billing system, cause the billing systemto bill the subscriber ID.
 12. The method of claim 11 further comprisingenabling the QoE network appliance to change a QoE requirement, whichresults in a price change for the changed QoE requirement.
 13. A methodfor measuring quality of experience (QoE) satisfaction for anapplication associated with one or more subscribers that are accessing aCSP network, the method comprising: executing the application on amobile device that is communicatively coupled to the CSP network;identifying a quality of experience (QoE) requirement for theapplication associated with the one or more subscribers accessing theCSP network, in which the QoE requirement for the application includes aQoE latency requirement, a QoE bandwidth requirement, and a QoE packetloss rate requirement; identifying a radio access network (RAN) dataset; identifying a core network (CN) data set that includes a networkdata analytics function (NWDAF) data set; gathering the RAN data set,the CN data set and the NWDAF data set with an edge-collection modulethat communicates with a QoE network appliance, which includes a corecompute and storage network component; associating, at the QoE networkappliance, the RAN data set, the CN data set, and the NWDAF data setwith a QoE latency measurement, a QoE bandwidth measurement, and a QoEpacket loss rate measurement; determining, at the QoE network appliance,the QoE requirement with the RAN data set, the CN data set, and theNWDAF data set; generating, at the QoE network appliance, a measured QoEscore with the RAN data set, the CN data set, and the NWDAF data setwherein the measured QoE score is associated with the latencymeasurement, the bandwidth measurement, and the packet loss ratemeasurement; determining when the measured QoE score satisfies the QoErequirement by comparing the latency requirement, the bandwidthrequirement, and the packet loss rate requirement associated with theQoE requirement, with the latency measurement, the bandwidthmeasurement, and the packet loss rate measurement associated with themeasured QoE score; and billing a subscriber ID when a charging functiondetermines the measured QoE satisfies the QoE requirement and having thecharging function, which is communicatively couple to a billing system,cause the billing system to bill the subscriber ID.
 14. The method ofclaim 13 further comprising enabling the QoE network appliance to changea QoE requirement, which results in a price change for the changed QoErequirement.
 15. The method of claim 13 wherein edge-based collectionmodule reduces a volume of data from at least one of the RAN data set,the NWDAF data set and the CN data set before communicating the reducedvolume of data to the QoE network appliance.
 16. The method of claim 13further comprising forecasting, at the QoE network appliance, a perdevice network load for a scaling model with at least one of the RANdata set, the NWDAF data set and the CN data set.
 17. The method ofclaim 13 further comprising forecasting, at the QoE network appliance, aper application network load for a scaling model with the DPI data set,the RAN data set, the NWDAF data set and the CN data set.
 18. The methodof claim 13 further comprising forecasting, at the QoE networkappliance, a per location network load for a scaling model with the DPIdata set, the RAN data set, the NWDAF data set and the CN data set. 19.The method of claim 13 further comprising determining a CSP networkpolicy; determining a reduced network performance with the measured QoEscore at an impacted area; and enabling the CSP network to alert atleast one mobile device of the reduced network performance.
 20. Themethod of claim 13 further comprising determining a CSP network policy;and changing the CSP network policy based on the measured QoE for thesubscriber ID at a particular cell.
 21. A system for measuring qualityof experience (QoE) satisfaction for an application associated with oneor more subscribers accessing a CSP network, the system comprising: amobile device executing the application and the mobile devicecommunicatively coupled to the CSP network; a quality of experience(QoE) requirement corresponding to the application associated with oneor more subscribers that accesses the CSP network, in which the QoErequirement for the application includes a QoE latency requirement, aQoE bandwidth requirement, and a QoE packet loss rate requirement; aradio access network (RAN) data set is gathered with an edge-collectionmodule; a core network (CN) data set that is gathered with anedge-collection module, wherein the CN data set includes a network dataanalytics function (NWDAF) data set; a QoE network appliance, whichincludes a core compute and storage network component, that receives theRAN data set, the CN data set and the NWDAF data set from theedge-collection module; the QoE network appliance associating the RANdata set, the CN data set, and the NWDAF data set with a QoE latencymeasurement, a QoE bandwidth measurement, and a QoE packet loss ratemeasurement; the QoE network appliance determining the QoE requirementwith the RAN data set, the CN data set, and the NWDAF data set; the QoEnetwork appliance generates a measured QoE score with the RAN data set,the CN data set, and the NWDAF data set wherein the measured QoE scoreis associated with the latency measurement, the bandwidth measurement,and the packet loss rate measurement; wherein the measured QoE scoresatisfies the QoE requirement by comparing the latency requirement thebandwidth requirement, and the packet loss rate requirement associatedwith the QoE requirement, with the latency measurement, the bandwidthmeasurement, and the packet loss rate measurement associated with themeasured QoE score; and a subscriber ID is billed when a chargingfunction determines the measured QoE satisfies the QoE requirement andhaving the charging function, which is communicatively couple to abilling system, cause the billing system to bill the subscriber ID. 22.The system of claim 21 further comprising a deep packet inspection (DPI)data set, wherein the DPI data set is selected from the open systeminterconnection (OSI) group consisting of a network layer, a transportlayer, a session layer, a presentation layer, and an application layer;and wherein the DPI data set is associated with the latency measurement,the bandwidth measurement, and the packet loss rate measurement; whereinthe DPI data set is also used to determine when the measured QoE scoresatisfies the QoE requirement.
 23. The system of claim 22 whereinedge-based collection module reduces a volume of data from at least oneof the DPI data set, the RAN data set, the NWDAF data and the CN dataset before communicating the reduced volume of data to the QoE networkappliance.
 24. The system of claim 22 wherein the DPI data setdetermines an optimal bandwidth measurement for a particularapplication, in which the optimal bandwidth measurement is associatedwith the measured QoE score.
 25. The system of claim 22 wherein the DPIdata set determines an optimal latency measurement for the particularapplication, in which the optimal latency measurement is associated withthe measured QoE score.
 26. The system of claim 21 wherein the QoEnetwork appliance forecasts a per device network load for a scalingmodel with at least one of the RAN data set, the NWDAF data set and theCN data set.
 27. The system of claim 21 wherein the QoE networkappliance forecasts a per application network load for a scaling modelwith the DPI data set, the RAN data set, the NWDAF data set and the CNdata set.
 28. The system of claim 21 wherein the QoE network applianceforecasts a per location network load for a scaling model with the DPIdata set, the RAN data set, the NWDAF data set and the CN data set. 29.The system of claim 21 further comprising a CSP network policy, in whicha reduced network performance is determined with the measured QoE scoreat an impacted area, and the CSP network alerting at least one mobiledevice of the reduced network performance.
 30. The system of claim 21further comprising a CSP network policy that changes based on themeasured QoE for the subscriber ID at a particular cell.